The invention relates to techniques for rapidly, accurately producing an in-focus image of an object, or a cross-section thereof, wherein the effect of light signals from out-of-focus foreground and/or background light sources are mostly eliminated. Confocal microscopy is finding many applications in life sciences, ophthalmology, industrial inspection, and semiconductor line-width metrology. This is because of the unique three-dimensional (3D) imaging capability of these instruments.
There are two general approaches for determining the volume properties of 3-D microscopical specimens. Such approaches are based on conventional microscopy and confocal microscopy. Generally, the conventional microscopy approach requires less time to acquire the data but more time to process the data for a 3-D image, compared to the confocal microscopy approach. The wide field method (WFM) (D. A. Agard and J. W. Sedat, Proc. Soc. PhotoOpt. Instrum. Eng., Vol. 264, p. 110, 1980; D. A. Agard, et al., Anal. Biochem., Vol. 111, p. 257, 1981; D. A. Agard, et al., Methods Cell Biol., Vol. 30, p. 353, 1989; D. A. Agard, Annu. Rev. Biophys. Bioeng., Vol. 13, p. 191, 1984; Y. Hiroaka, et al., Sci., Vol. 238, p. 36, 1987; W. Denk, et al., Sci., Vol. 248, p. 73, 1990) uses a conventional microscope to sequentially acquire a set of images of adjacent focus planes throughout the volume of interest. Each image is recorded using a cooled charge-coupled device (CCD) image sensor (J. Kristian and M. Blouke, Sci. Am., Vol. 247, p. 67, 1982) and contains data from both in-focus and out-of-focus image planes.
In a conventional imaging system, when a part of the object to be imaged is axially displaced from its best focus location, the image contrast decreases but the brightness remains constant so that displaced, unfocused parts of the image interfere with the view of focused parts of object.
If the system's point-spread function is known and images are obtained for each independent section of the object, known computer algorithms can be applied to such images to effectively remove the signal contributed by the out-of-focus light and produce images that contain only in-focus data. Such algorithms are of several distinct types, are referred to as "computer deconvolutions", and generally require expensive computer equipment and considerable computing time.
The technique of laser computed tomography is implemented using a conventional microscope. The system discussed by S. Kawata, et al., Appl. Opt., Vol. 29, p. 3805, (1990) is based on a principal that is closely related to the technique of X-ray computed tomography, but uses 3-D volume reconstruction rather than 2-D slice reconstruction. Projected images of a thick 3-D sample are collected with a conventional transmission microscope modified with oblique illumination optics, and the 3-D structure of the interior of the sample is reconstructed by a computer. Here, the data is acquired in a time short compared to that required to process data for a 3-D image. In one experiment by Kawata et al., op. cit. , the 80.times.80.times.36-voxel reconstruction required several minutes to collect all projections and send them to a minicomputer. Approximately thirty minutes then were required for digital reconstruction of the image, in spite of utilizing a vector processor at a speed of 20 million floating point operations per second (MFLOPS).
In a conventional point or pinhole-confocal microscope, light from a point source is focused within a very small space, known as a spot. The microscope focuses light reflected from, scattered by, or transmitted through the spot onto a point detector. In a reflecting point-confocal microscope the incident light is reflected or back-scattered by that portion of the sample in the spot. Any light which is reflected or back-scattered by the sample outside of the spot is not well focused onto the detector, thus it is spread out so the point detector receives only a small portion of such reflected or back-scattered light. In a transmitting point-confocal microscope, incident light is transmitted unless it is scattered or absorbed by that portion of the sample in the spot. Generally, the point source and point detector are approximated by placing masks containing a pinhole in front of a conventional light source and conventional detector, respectively.
Similarly, in a conventional slit-confocal microscope system, light from a line source is focused into a very narrow elongated space, which is also known as a spot. The slit-confocal microscope focuses light reflected from, scattered by or transmitted through the spot onto a line detector. The line source and line detector can be approximated using a mask with a slit in front of a conventional light source and row of conventional detectors, respectively. Alternately, a line source can be approximated by sweeping a focused laser beam across the object to be imaged or inspected.
Since only a small portion of the object is imaged by the confocal microscope, either the object to be imaged must be moved, or the source and detector must be moved, in order to obtain sufficient image data to produce a complete 2-D or 3-D view of the object. Previous slit-confocal systems have moved the object linearly in a direction perpendicular to the slit to obtain successive lines of 2D image data. On the other hand, point-confocal systems having only one pinhole have to be moved in a 2-D manner in order to acquire 2-D image data and in a 3-D manner in order to acquire a 3-D set of image data. The raw image data are typically stored and later processed to form a 2-D cross-section or a 3-D image of the object that was inspected or imaged. Because of the reduced sensitivity to the out-of-focus images, the processing operation is considerably simpler in comparison to that required when processing data obtained in conventional microscopy approach.
In a system known as the Tandem Scanning Optical Microscope (TSOM), a spiral pattern of illumination and detector pinholes are etched into a Nipkow disk so, as the disk rotates, the entire stationary object is scanned in 2-D. In terms of the optical processing, the TSOM is basically a single point confocal microscope with a means for efficiently scanning a 2-D section one point at a time.
Examples of two techniques implemented to reduce the amount of scanning required to obtain a 2-D image with a confocal arrangement are found in the work of H. J. Tiziani and H. -M. Uhde, Appl. Opt., Vol. 33, p. 567, (1994) and in the patent of Kerstens et al. (U.S. Pat. No. 5,248,876 issued September 1993). The microlens-array confocal arrangement of Tiziani and Uhde op. cit. has out-of-focus image discrimination that is the same as using a multipin-hole source and detector in a confocal configuration. Such a system allows for a number of points to be examined simultaneously but at a compromise in discrimination against out-of-focus images. The higher the density of microlenses, the poorer the ability of the system to discriminate against out-of-focus images, and consequently, an increase in complexity and cost of the computer deconvolutions required to produce a 3-D image. Further, the Tiziani and Uhde op. cit. system has serious limitations in axial range. This range cannot exceed the focal length of the microlens, which is proportional to the diameter of the microlens for a given numerical aperture. Therefore, as the density of the microlenses is increased, there is an associated decrease in the permitted axial range.
The Kerstens et al. op. cit. system incorporates a number of pinholes and matching pinpoint detectors in a confocal arrangement to allow for a number of points to be examined simultaneously. However, as noted in the preceding paragraph, this gain is at a compromise in discrimination against out-of-focus images and as a result an increase in complexity and cost of required subsequent computer deconvolutlons. The higher the density of pinholes, the poorer the ability of the system to discriminate against out-of-focus images. The highest discrimination would be achieved when using only one pinhole.
Application of confocal microscopes to inspection of electronics was suggested in Sapf et al., Microelectronic Engineering, Vol. 5, p. 573 (1986) and Lindow, et al., SPIE, Vol. 565, p. 81 (1985). The axial discrimination provided by confocal systems make them useful in the semi-conductor manufacturing environment. For example, such systems could provide for improved inspection of height dependent features such as delamination, blisters, and thickness of structures and coatings. However, there are some problems associated with using confocal imaging systems for inspection of electronics. For example, single pinhole systems require too much time for scanning the object in two directions. Optical systems for scanning a laser beam over the object are too complex; and the spinning disk approach used in the previous Tandem Scanning Microscope resulted in alignment and maintenance problems.
Thus, none of the prior systems can be configured for rapid and/or reliable three dimensional tomographic imaging, especially in the field of inspection or imaging.
The number of different depth slices required (and therefore the amount of image data collected) depends upon the range of height that must be measured, and also upon the desired height resolution and performance of the optical system. For typical electronics inspection, images of 10 to 100 different depth slices would be required. Furthermore, data in several color bands may be required to differentiate materials. In confocal imaging systems, a separate 2-D scan is required for each desired elevation. If data for multiple color bands is desired, then multiple 2-D scans at each elevation are required. By shifting the focus level, similar data can be obtained from adjacent planes and a 3-D intensity data set can be acquired.
Although the confocal approach is more straightforward and works better, for example in confocal fluorescence work, when the concentration of stained structure is high, the conventional microscopy approach still has several practical advantages. The most important of these is that the latter can utilize dyes that are excited in the ultraviolet (UV) range and these often seem more robust and efficient than those excited in the visible range. Although, a UV laser can be incorporated as the light source of a confocal microscope (M. Montag, et al., J. Microsc (Oxford), Vol. 163 (Pt. 2), p. 201, 1991; K. Kuba, et al., Neurosci. Res., Vol. 10, p. 245, 1991; C. Bilton, et al., J. Microsc., Vol. 196, p. 15, 1993), or UV dyes can be excited with infrared (IR) light using the "two photon" technique (W. Denk, et al., op. cit.), these techniques involve considerable expense and practical difficulty.
Furthermore, the cooled CCD detectors used in conventional microscopy systems collect the data in parallel rather than serially, as does the photomultiplier (PMT) in a confocal microscopy system. As a result, if the CCD can be made to read out more rapidly without degrading its performance, the 3-D data recording rate of the conventional microscopy system may prove to be significantly higher than that of the confocal microscopy system, even though the time needed for computer deconvolution computations means that there might be an additional delay before the data could be actually viewed as a 3-D image.
The signal-to-noise ratio in relation to statistical accuracy must also be considered when making a choice between a CCD detector used to record in parallel a 2-D data array and a slit or pinhole confocal microscope. The well capacity of a 2-D CCD pixel is of the order of 100,000 electrons. This limits the statistical accuracy that can be achieved in a single exposure as compared to that achievable with other photoemissive detectors such as PMT's or photovoltaic devices. Consequently, for those applications where the out-of-focus background contributions are significantly larger than the in-focus image signals, consideration of the signal-to-noise ratio may lead to the conclusion that a 1-D parallel recording of data in a slit confocal microscope will perform better than a 2-D recording of data in a standard microscope configuration or a point by point recording of data in a single pinhole confocal microscope will perform better than a 1-D parallel recording of data in a slit confocal microscope, all other considerations being equal.
When the consideration of statistical accuracies as measured by the signal-to-noise ratios influences the selection of a system such as a slit confocal microscope over a standard microscope, or a single pinhole confocal microscope over a slit confocal microscope, the residual signals from the out-of-focus images for the system chosen can be comparable to or larger than the in-focus signals. Such is the case for example when examining deep into biological samples at optical wavelengths where scattering of optical radiation dominates over absorption. In this case, one is left with the need for a lengthy computer deconvolution, i.e. long compared to the time required to acquire the data. Note that this is in general true for the single pinhole confocal microscope as well as the slit confocal microscope when looking for an in-focus image signal that is much smaller than the residual out-of-focus image signals.
Although it is easier to accurately digitize the signal from a CCD detector than from a PMT (J. B. Pawley, Scanning, Vol. 13, p. 184, 1991), the PMT is a single device that can be accurately characterized, whereas the CCD is actually a large array of discrete detectors and additional noise is associated with correcting for the pixel-to-pixel variations in sensitivity and offset that characterize its operation (Y. Hioraka, et al., op. cit.; J. E. Wampler and K. Kutz, Methods Cell Biol., Vol. 29, p. 239, 1989; Z. Jericevic, et al., Methods Cell Biol., Vol. 30, p. 47, 1989).
It should be noted that the above distinction between the photodetectors used in the two methods of 3-D microscopy should not be considered to be complete, because the cooled CCD detector is the most suitable photodetector for those confocal microscopes that accomplish the scanning function by using holes in a spinning disk (M. Petran, et al., J. Opt. Soc. Am., Vol 58, p.661, 1968; G. Q. Xiao, et al., Appl. Phys. Letts., Vol. 53, p. 716, 1988).
D. Hickman, et al., Opt. Letts., Vol. 13, p. 898 (1980) separated a highly coherent light beam generated by a laser from a non-coherent background (non-laser-generated background) by relying on the differences in temporal coherence to separate the desired and background signals. Their technique is not applicable to the more commonly encountered non-coherent sources or when the background is generated by scattering of the coherent light bear, itself.
FIG. 6 of Hill (1983 Final Report, ONR Contract No. 00014-81-C-0794) shows a system where the images of two telescopes are combined by the use of a beam splitter system. A variant of this type has been used by Currie et al., Astrophys. J., Vol. 187, p. 131 (1974) to recover the diffraction limit of the telescope while observing through the earth's atmosphere. In that work, the properties of the autocorrelation function of the electronic signals S.sub.1 and S.sub.2 from the two detectors D.sub.1 and D.sub.2, respectively, detecting the two light beams out of the beam splitter were used to obtain the diffraction-limited performance of the instrument. However, any background at all greatly degrades the ability of the Currie, et al., op. cit. application to achieve the diffraction-limited performance.
An early variant of the concept shown in FIG. 6 of Hill, op. cit. was applied to radio astronomy by M. Ryle, Proc. Roy. Soc. of London, Vol. A211, p. 351 (1951). Ryle op. cit. used the technique to separate the radio signal of a point source from an extended uniform source by introducincg an alternating length of time delay line between the electronic signals received from two separate radio antenna detector units. The signals are combined in this case electronically instead of by using a beam splitter system. The degree to which the background discrimination can be achieved depends on how accurately the responsivity of the two respective detectors are known and compensated in the subsequent signal processing, on how immune the electronics in the variable delay-line leg are to changes in the delay-line length, and on the degree of narrow band filtering of the radio signals that is used. If detector arrays are used for D.sub.1 and D.sub.2, then there must be an independent alternating length of delay line for each complementary pair of detectors in the detector arrays. Also, this technique is not applicable to higher-frequency electromagnetic signals, such as those found in the visual range.
Hill and Oglesby in Hill op. cit. noted that a system such as shown in FIG. 6 of the Hill op. cit. report could be useful in measuring the amount of scattered light in the line-of-sight. In that system, the difference of the normalized electronic signals S.sub.1 from detector D.sub.1 and S.sub.2 from detector D.sub.2 contains only contributions for the common field of view (FOV) of the two telescopes, where the point source response functions for S.sub.1 and S.sub.2 are used for normalization. Application of each of the three above described variants is limited to highly localized light point sources. The filters Q.sub.1 and Q.sub.2 are used in order to meet the narrow wavelength passband requirements for those applications where the source itself is not narrowband.
What is needed is a system that combines the parallel data acquisition feature of a conventional microscopy system, reduced sensitivity to pixel-to-pixel variations in multidetector systems such as the CCD, the reduced sensitivity of the image data to out-of-focus images inherent in confocal microscopy systems, and the reduced requirement of computer deconvolutions associated with reduced sensitivity to out-of-focus images.